Observations

• We seem to spawn a lot of Tokio worker threads which Tokio's runtime needs to schedule (every transaction is spawned its own Tokio worker thread. This is unbounded)
• In archive_event we performed update_bundle_history and update_tx_by_hash sequentially and blocked on each operation resulting in the duration being upwards of 600ms.
• To the first point, we might be getting rate limited by S3

Changes

Create a fixed number of workers (80) and do the updating in parallel

• If there are 350 txs / block (looking at Mainnet), then that's approx 175 txs/sec
• In the past 4 hrs, update_bundle_history worst case took ~500ms and same for update_tx_by_hash. we can do this updating in parallel
• So the number workers needed would be ~88 workers (175 events/sec * 0.5sec/event)

A fixed worker pool avoids resource exhaustion and provides backpressure via a bounded channel

Results

• Over 10k transactions sent on Sepolia Alpha, peak archive_event_duration was 3.02ms, previous was 5.27ms.

TODO:

• fix test_userop_kafka_publisher_reader_integration test failing (nvm docker wasnt running)